A storage system is a computer that provides storage service relating to the organization of information on writeable persistent storage devices, such as memories, tapes or disks. The storage system is commonly deployed within a storage area network (SAN) or a network attached storage (NAS) environment. When used within a NAS environment, the storage system may be embodied as a file server including an operating system that implements a file system to logically organize the information as a hierarchical structure of directories and files on, e.g. the disks. Each “on-disk” file may be implemented as a set of data structures, e.g., disk blocks, configured to store information, such as the actual data for the file. A directory, on the other hand, may be implemented as a specially formatted file in which information about other files and directories are stored.
The file server, or filer, may be further configured to operate according to a client/server model of information delivery to thereby allow many client systems (clients) to access shared resources, such as files, stored on the filer. Sharing of files is a hallmark of a NAS system, which is enabled because of semantic level of access to files and file systems. Storage of information on a NAS system is typically deployed over a computer network comprising a geographically distributed collection of interconnected communication links, such as Ethernet, that allow clients to remotely access the information (files) on the file server. The clients typically communicate with the filer by exchanging discrete frames or packets of data according to pre-defined protocols, such as the Transmission. Control Protocol/Internet Protocol (TCP/IP).
In the client/server model, the client may comprise an application executing on a computer that “connects” to the filer over a computer network, such as a point-to-point link, shared local area network, wide area network or virtual private network implemented over a public network, such as the Internet. NAS systems generally utilize file-based access protocols; therefore, each client may request the services of the filer by issuing file system protocol messages (in the form of packets) to the file system over the network. By supporting a plurality of file system protocols, such as the conventional Common Internet File System (CIFS), the Network File System (NFS) and the Direct Access File System (DAFS) protocols, the utility of the filer may be enhanced for networking clients.
A SAN is a high-speed network that enables establishment of direct connections between a storage system and its storage devices. The SAN may thus be viewed as an extension to a storage bus and, as such, an operating system of the storage system enables access to stored information using block-based access protocols over the “extended bus”. In this context, the extended bus is typically embodied as Fibre Channel (FC) or Ethernet media adapted to operate with block access protocols, such as Small Computer Systems Interface (SCSI) protocol encapsulation over FC (FCP) or TCP/IP/Ethernet (iSCSI). A SAN arrangement or deployment allows decoupling of storage from the storage system, such as an application server, and some level of storage sharing at the application server level. There are, however, environments wherein a SAN is dedicated to a single server.
It is advantageous for the services and data provided by a storage system to be available for access to the greatest degree possible. Accordingly, some computer systems provide a plurality of storage systems organized as nodes of a cluster, with a property that when a first storage system fails, a second storage system is available to take over and provide the services and the data otherwise served by the first storage system. In particular, when the first storage system fails, the second storage system in the cluster assumes the tasks of processing and handling any data access requests normally processed by the first storage system. One such example of a cluster configuration is described in U.S. patent application Ser. No. 09/625,234 entitled NEGOTIATING TAKEOVER IN HIGH AVAILABILITY CLUSTER by Samuel M. Cramer, et al.
In a typical cluster environment, the various storage systems are interconnected by one or more cluster interconnects that are typically implemented using Virtual Interface (VI) layered on Fibre Channel (FC-VI) over a physical FC transport medium. Virtual Interfaces are further described in Virtual Interface Architecture Specification, Version 1.0 by Compaq Computer Corp. et al., the contents of which are hereby incorporated by reference. A cluster interconnect is illustratively a direct network connection linking two or more storage systems in a cluster environment. The storage systems utilize the VI-protocol, including its native remote direct access (RDMA) capabilities, during normal cluster operation over the cluster interconnect. By “native” it is meant that the transport protocol, e.g., FC-VI, supports the desired functionality, e.g. RDMA, without modifications.
Executing on each of the various storage systems typically is a storage operating system that manages the storage systems and may implement file system semantics. The storage operating system must be robust to prevent data loss from error conditions arising while executing on a storage system. As such, during development, the storage operating system must be thoroughly tested before being released for use in actual data serving environments. One method for performing such a test during development is to simulate the storage operating system on a host computer that is executing its own operating system. This storage operating system simulator entails portions of the storage operating system executing as a user-level application on the host computer. In this testing environment, disks associated with a storage system are typically represented by files on the host computer's file system. Thus, write operations from a disk driver module of the storage operating system are converted to file-level operations by the host's operating system. Similarly, network protocol communications, such as by TCP/IP, are passed from the storage operating system to the host computer's operating system for processing.
A noted disadvantage of such a testing configuration is that the protocol stack of the simulated storage operating system is not tested as the processes of the stack are “offloaded” to the host computer's operating system. To simulate a clustering environment, two separate instantiations of the simulated storage operating system may be executed as separate processes on the host computer. However, in such configurations, the cluster interconnect adapter is typically represented by a shared memory segment that is used as a mailbox between the two simulated storage operating systems. Unfortunately, this high-level of abstraction eliminates the protocol software stack that drives the cluster interconnect. Thus, this interconnect communication protocol stack can only be tested and/or debugged on a live storage system cluster, which eliminates the cost savings and control of testing on a storage operating system simulator.